Not only Linux but several other file systems don't need defragging (most enterprise systems). I'm talking about several generations of file systems. FAT and NTFS are what you could call 2nd (16 bit, no access control, later on directory structures) and 3rd generation file systems (32 bit, access control etc.) but still stems from the 80's and 90's.
Ext3 is a 4th generation file system where people started using journaling (keeping track of changes that weren't finished yet so it can roll it back when the system crashes instead of doing a disk check to find loose ends, databases and b-trees to keep file information) which got much better performance for really large file systems. Ext4 and ZFS are current generation file systems (128 bit) that scale to several Exabytes if not more and use all the things we have learned over the years and improves them even further.
As said, the little bit of space is in the simplest of advanced file systems (ext3) a block (off course depending on formatting but usually 4 kbyte). It would have been 'wasteful' for the 1980's where a hard drive was 10MB and cost several hundreds but these days, do you really think you're missing 4 kbyte/file (~40MB out of your average 1TB disk)? There is always a trade-off to be made between performance and being efficient, there are file systems specifically meant for embedded systems (where even a traditional allocation table would be too much). The reasoning of the creators was that, ok, if you have small changes and additions to files, you shouldn't be penalized for that in read performance by having the disk thrashing all over the place to read a single file. On the other hand, for large files, additions would be either not existent or be in itself large additions so it is not really a problem to have a large file divided in two pieces since that requires just 1 interruption. A program can also 'state' how much space a file needs and thus an area on the disk is reserved as a single large block regardless of whether it's written to or not.
FAT and NTFS systems have a similar problem where they can only allocate blocks for a single file, thus if you have a 5kbyte file and your formatted at 4kbyte/block, you would break up the file and write 4kb + 1kb but actually use the space of 8kb but since FAT can't append data it would for the next 5kbyte use yet another 2 blocks somewhere else on the disk and use now 16kb for your 10kb file. Defragmenting brings those blocks together. Fragmentation still happens in ext3 but it is way less severe and does not significantly impact performance.
The intent log in ext4 and zfs (and some other recent file systems) works by writing to a specific area on disk (this is oversimplified, the technical details differ per implementation and I don't want to look them up right now) or in some cases, solid state disks or battery backed-up memory the small changes and then later (when it gets flushed, the file gets closed or when the system is idle) re-writing the entire file in order to a designated area of the disk. Thus your file is no matter how fast or slow you write to it (ie. if you're downloading pieces of the internet) always rewritten in the same area of the disk. ZFS actually can use variable block sizes (up to 128kb) so you're not 'wasting' a lot of space and you can also read large files much faster (as you give 1 command and you get up to 128kb instead of 32 commands getting 4kb each with fixed block sizes).
The adaptive caching since ext3, 4 etc. and limited in Windows 7 too is indeed using system memory (and in case of ZFS you can also use Solid-State Drives as a secondary cache) and why not - you have what 2-4GB in your machine, your current applications only need ~500MB, why not read out the rest of your files while the system is busy doing something else so when you need it, you don't need to wait on the disk to thrash around, you just read from memory thereby negating any fragmentation issues as the reads happen when the system is processing previous data. If the memory is under pressure, it will evict the least used pieces first so you always have the most used pieces in memory.
Also intelligent systems like ZFS and certain RAID controllers also (ab)uses the on-disk buffers - they'll command the hard drive to fetch actually a little more data than necessary (the adjacent blocks) but won't actually read them so that if the system decides to ask for them anyway, they can be read from the disk buffer but on the other hand, the bandwidth is not wasted trying to read and process them.
Microsoft has been attempting to come up with an answer (WinFS has been promised since Windows 2000 but is still not working in Windows 8) for the fragmentation issue of NTFS/FAT and even Apple has made advancements in it's HFS+ system. SSD's would be the easiest answer (fragmentation doesn't matter) but the performance would still be atrocious if you go into the details. That's why nobody uses Windows as a storage platform.
In the above examples about caching etc. you have to think - how much data do you actually use. I have ~50TB at work and the home directories of ~20 simultaneous users and ~500GB of read cache. There is only about 200GB of user data that gets most commonly read, yeah we have a lot of stuff but we never, ever read it. How many times do you look at all your pictures or listen to all your music?